{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "b7f27c9d-d62d-4bf4-aea7-172072cf07a8",
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "%load_ext autoreload\n",
        "%autoreload 2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "9b302ee7",
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "from torch.utils.data import DataLoader\n",
        "from torch.optim.lr_scheduler import MultiStepLR\n",
        "import torchvision\n",
        "import matplotlib.pyplot as plt\n",
        "import dataset\n",
        "from model import ResNet18YOLOv1\n",
        "from loss import YOLOv1Loss\n",
        "from tqdm import tqdm\n",
        "import numpy as np"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4ecbed67-d7f8-4c72-9cfa-429f38b82556",
      "metadata": {
        "tags": []
      },
      "source": [
        "# About\n",
        "This is an implementation of YOLOv1 from ***You Only Look Once: Unified, Real-Time Object Detection by Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Firhadi.*** Object detection is figuring out what objects are in an image and where they are. Another way to look at this problem is how can we write a computer program that draws bounding boxes around objects and predicts what kind of objects they are. YOLO solves this problem and does it super fast, like state of the art fast!\n",
        "\n",
        "Let's talk about R-CNN, the predecessor to YOLO. It proposed regions, ran a classifier on every region, and did some post-processing to produce the final result. In simple language this translates to:\n",
        "1. Lemme draw a lot of bounding boxes where I think objects are\n",
        "2. Lemme figure out what are in the bounding boxes I drew\n",
        "3. Ok, I drew too many bounding boxes, lemme remove most of them and keep the important ones\n",
        "\n",
        "This is a lot of steps. What YOLO does instead is ***unified detection***. Unified detection combines the different components of object detection (where are the objects and what kind of objects are they) into one Convolutional Neural Network. You give it an image and in one swoop, it tells you exactly that.\n",
        "\n",
        "Here's how it does it:\n",
        "1. Divide the image into a SxS grid\n",
        "2. Each cell in the grid predicts B bounding boxes and C class probabilities (what it thinks the object is)\n",
        "\n",
        "We represent bounding boxes with 5 numbers: x, y, w, h, p.\n",
        "- (x, y): center of the bounding box\n",
        "- w: width\n",
        "- h: height\n",
        "- p: confidence (a measure of how confident we are that this box captures an object and matches the ground truth)\n",
        "\n",
        "Accordingly, YOLOv1 produces a SxSx(5B+C) tensor. Each cell predicts B bounding boxes, how do we choose which one is the \"true\" predictor? How do we measure how good our bounding box and classification predictions are? \n",
        "\n",
        "We check which bounding box has the greatest overlap (IOU: Intersection Over Union) with the ground truth and choose that one as a predictor. We use this loss function to measure the \"goodness\" of our predictions:\n",
        "\n",
        "![yolo loss function](https://i.stack.imgur.com/IddFu.png)\n",
        "\n",
        "On a high level, it is the squared error between our prediction and the ground truth. "
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2189430c-7b31-498f-8498-e5254ba79dc4",
      "metadata": {
        "tags": []
      },
      "source": [
        "# PASCAL VOC 2007 Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9ff97fac-bdec-4ce3-b2d6-a4d6a7945642",
      "metadata": {},
      "source": [
        "PASCAL VOC Detection Dataset contains annotated images with 20 labelled classes and bounding boxes. There are 2,501 images in the training set, 2,510 images in the validation set, and 4,952 images in the test set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "5525f8a3-d645-45ea-b44c-9f1ca9ab6c9b",
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "# original dataset\n",
        "pascal_voc_train = torchvision.datasets.VOCDetection(\n",
        "    root=\"data\",\n",
        "    year=\"2007\",\n",
        "    image_set=\"train\",\n",
        "    download=False\n",
        ")\n",
        "\n",
        "pascal_voc_val = torchvision.datasets.VOCDetection(\n",
        "    root=\"data\",\n",
        "    year=\"2007\",\n",
        "    image_set=\"val\",\n",
        "    download=False\n",
        ")\n",
        "\n",
        "pascal_voc_test = torchvision.datasets.VOCDetection(\n",
        "    root=\"data\",\n",
        "    year=\"2007\",\n",
        "    image_set=\"test\",\n",
        "    download=False\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "11520045",
      "metadata": {
        "tags": []
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "TRANSFORMING PASCAL VOC\n",
            "TRANSFORMING PASCAL VOC\n",
            "TRANSFORMING PASCAL VOC\n"
          ]
        }
      ],
      "source": [
        "# augment dataset for YOLOv1: resize and normalize image and convert bounding boxes from annotations to tensors\n",
        "voc_train = dataset.PascalVOC(pascal_voc=pascal_voc_train)\n",
        "voc_val = dataset.PascalVOC(pascal_voc=pascal_voc_val)\n",
        "voc_test = dataset.PascalVOC(pascal_voc=pascal_voc_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "id": "1abb4f1f",
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "BATCH_SIZE = 64"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "id": "a8674b35",
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "# train_dataloader = DataLoader(voc_train, batch_size=BATCH_SIZE, shuffle=True)\n",
        "# val_dataloader = DataLoader(voc_val, batch_size=BATCH_SIZE, shuffle=True)\n",
        "# test_dataloader = DataLoader(voc_test, batch_size=BATCH_SIZE, shuffle=True)\n",
        "\n",
        "train_dataloader = DataLoader(voc_train, batch_size=BATCH_SIZE, shuffle=False)\n",
        "val_dataloader = DataLoader(voc_val, batch_size=BATCH_SIZE, shuffle=False)\n",
        "test_dataloader = DataLoader(voc_test, batch_size=BATCH_SIZE, shuffle=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b831392b-941b-4c64-86fa-a23bf3c558f5",
      "metadata": {
        "tags": []
      },
      "source": [
        "# Training"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "bf5f716d-75e5-4367-9fdf-fe1843843bd5",
      "metadata": {
        "tags": []
      },
      "source": [
        "## Device"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "id": "5c323bd4-63e6-441d-bd8e-65b404122ba3",
      "metadata": {
        "tags": []
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "device(type='cuda')"
            ]
          },
          "execution_count": 31,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "DEVICE = \"cpu\"\n",
        "\n",
        "if torch.cuda.is_available():\n",
        "    DEVICE = torch.device(\"cuda\")\n",
        "if torch.backends.mps.is_available() and torch.backends.mps.is_built():\n",
        "    DEVICE = torch.device(\"mps\")\n",
        "\n",
        "DEVICE"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5065e90d-ef48-4244-92e9-2553e44518ce",
      "metadata": {
        "tags": []
      },
      "source": [
        "## Hyperparameters\n",
        "- S: dimensions of SxS grid\n",
        "- B: number of bounding boxes predicted per cell\n",
        "- C: number of classes"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "id": "2f9313e8-6e96-4d4a-8dd6-98d56df27ad4",
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "S = 7\n",
        "B = 2\n",
        "C = 20\n",
        "lambda_coord = 5.0\n",
        "lambda_noobj = 0.5"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "254ef8df-55aa-4f2d-8a6b-805f5ed1b276",
      "metadata": {
        "tags": []
      },
      "source": [
        "## Model\n",
        "ResNet18 convolutional layers pretrained on ImageNet with 2 feedforward layers outputting a (N x S x S x (5B + C)) tensor."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "id": "8400c085-8bac-4b61-b13d-3b100a6f7c17",
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "yolo = ResNet18YOLOv1(S=S, B=B, C=C).to(DEVICE)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "45afc591-7cce-4066-903d-60e436c74b58",
      "metadata": {
        "tags": []
      },
      "source": [
        "## Loss + Optimizer\n",
        "![yolo loss function](https://i.stack.imgur.com/IddFu.png)\n",
        "\n",
        "We use stochastic gradient descent with a learning rate of 1e-3, weight decay (L2 regularization) of 0.0005, and momentum of 0.9."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "id": "d36287be-210a-452c-bf59-3874c2cb6681",
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "yolo_loss = YOLOv1Loss(S=S, B=B, C=C, lambda_coord=lambda_coord, lambda_noobj=lambda_noobj)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8cbbfadb-3e8d-468e-9695-0e6b67b65f20",
      "metadata": {},
      "source": [
        "## Train\n",
        "Train model with a learning rate of 1e-3 for the first few epochs, raise learning rate to 1e-2 and train for 75 epochs, then lower to 1e-3 for 30 epochs, and finally 1e-4 for 30 epochs.\n",
        "\n",
        "We train the network for about 135 epochs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "id": "cc42c578-7fd1-462a-b415-c45d8529573e",
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "def evaluate_loss(model, criterion, dataloader):\n",
        "    total_loss = 0\n",
        "    model.eval()\n",
        "    \n",
        "    with torch.no_grad():\n",
        "        for X, Y in dataloader:\n",
        "            X = X.to(DEVICE)\n",
        "            Y = Y.to(DEVICE)\n",
        "\n",
        "            pred = model(X)\n",
        "            loss = criterion(pred, Y)\n",
        "            total_loss += loss.item()\n",
        "            \n",
        "    N = len(dataloader)\n",
        "    # loss = total_loss / N\n",
        "    loss = total_loss\n",
        "    \n",
        "    return loss"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "id": "e95cd065-6e67-4363-b8ad-cd0c5d9831d4",
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "torch.cuda.empty_cache()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "id": "da10513c-17e8-473b-925b-8916ddaeb8d8",
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "EPOCHS = 135\n",
        "MOMENTUM = 0.9\n",
        "WEIGHT_DECAY = 0.0005\n",
        "\n",
        "def train_yolo(model, criterion, train_dataloader, val_dataloader):\n",
        "    pre_optimizer = torch.optim.SGD(model.parameters(), lr=1e-3, weight_decay=WEIGHT_DECAY, momentum=MOMENTUM)\n",
        "    optimizer = torch.optim.SGD(model.parameters(), lr=1e-3, weight_decay=WEIGHT_DECAY, momentum=MOMENTUM)\n",
        "    scheduler = MultiStepLR(optimizer,\n",
        "                            milestones=[75, 105],\n",
        "                            gamma=0.1)\n",
        "    \n",
        "    train_losses = []\n",
        "    val_losses = []\n",
        "\n",
        "    N = len(train_dataloader)\n",
        "\n",
        "    # train\n",
        "    for epoch in range(EPOCHS):\n",
        "        model.train()\n",
        "        total_loss = 0\n",
        "        lr = optimizer.param_groups[0][\"lr\"]\n",
        "        for i, (X, Y) in enumerate(tqdm(train_dataloader, leave=False, desc=f\"Epoch [{epoch+1}/{EPOCHS}]: lr={lr}\")):\n",
        "            X = X.to(DEVICE)\n",
        "            Y = Y.to(DEVICE)\n",
        "\n",
        "            pred = model(X)\n",
        "            \n",
        "            # print(pred)\n",
        "            loss = criterion(pred, Y)\n",
        "            # print(loss)\n",
        "            total_loss += loss.item()\n",
        "\n",
        "            # backprop\n",
        "            optimizer.zero_grad()\n",
        "            loss.backward()\n",
        "            optimizer.step()\n",
        "\n",
        "        train_loss = total_loss / N\n",
        "        train_losses.append(train_loss)\n",
        "\n",
        "        # update learning rate\n",
        "        scheduler.step()\n",
        "\n",
        "        # evaluate on validation set\n",
        "        val_loss = evaluate_loss(model, criterion, val_dataloader)\n",
        "        val_losses.append(val_loss)\n",
        "        \n",
        "        print(f\"Epoch [{epoch+1}/{EPOCHS}]: Train Loss={train_loss}, Val Loss={val_loss}\")\n",
        "    \n",
        "    return train_losses, val_losses"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "40116bcc-8aef-427f-8498-d0c43cc93077",
      "metadata": {
        "tags": []
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "                                                                        \r"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch [1/135]: Train Loss=13.076026654243469, Val Loss=8.242039775848388\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "                                                                        \r"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch [2/135]: Train Loss=7.555111837387085, Val Loss=7.0126620650291445\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "                                                                        \r"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch [3/135]: Train Loss=6.330504268407822, Val Loss=6.748044264316559\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "                                                                        \r"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch [4/135]: Train Loss=5.602589225769043, Val Loss=6.502433907985687\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "                                                                        \r"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch [5/135]: Train Loss=5.095652854442596, Val Loss=6.412951004505158\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "                                                                        \r"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch [6/135]: Train Loss=4.541189247369767, Val Loss=6.240354895591736\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "                                                                        \r"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch [7/135]: Train Loss=4.177050065994263, Val Loss=6.16488698720932\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "                                                                        \r"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch [8/135]: Train Loss=3.935801684856415, Val Loss=6.179985439777374\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "                                                                        \r"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch [9/135]: Train Loss=3.634955132007599, Val Loss=6.083581030368805\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "                                                                         \r"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch [10/135]: Train Loss=3.278478467464447, Val Loss=6.008610856533051\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "                                                                         \r"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch [11/135]: Train Loss=3.0551515400409697, Val Loss=5.977478551864624\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "                                                                         \r"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch [12/135]: Train Loss=2.8305365204811097, Val Loss=5.963375318050384\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Epoch [13/135]: lr=0.001:  42%|████▎     | 17/40 [00:13<00:18,  1.24it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch [25/135]: Train Loss=1.2844058126211166, Val Loss=5.891993165016174\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "                                                                         \r"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch [26/135]: Train Loss=1.2400614470243454, Val Loss=5.86917667388916\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "                                                                         \r"
          ]
        }
      ],
      "source": [
        "train_losses, val_losses = train_yolo(yolo,\n",
        "           yolo_loss,\n",
        "           train_dataloader=train_dataloader,\n",
        "           val_dataloader=val_dataloader)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "id": "53e91f29-a841-48b6-9f97-434f93cf92fa",
      "metadata": {
        "tags": []
      },
      "outputs": [
        {
          "ename": "NameError",
          "evalue": "name 'train' is not defined",
          "output_type": "error",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "Cell \u001b[0;32mIn[23], line 5\u001b[0m\n\u001b[1;32m      2\u001b[0m plt\u001b[38;5;241m.\u001b[39mxlabel(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mEpochs\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m      3\u001b[0m plt\u001b[38;5;241m.\u001b[39mylabel(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mLoss\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m----> 5\u001b[0m plt\u001b[38;5;241m.\u001b[39mplot(\u001b[38;5;28mrange\u001b[39m(EPOCHS), \u001b[43mtrain\u001b[49m)\n",
            "\u001b[0;31mNameError\u001b[0m: name 'train' is not defined"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkIAAAHFCAYAAAAe+pb9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAyxklEQVR4nO3de1xVVf7/8ffhfhlBRUW8hFjeGS0hCczMTErNsrGvliVq+p3IbspYac7kJWeYrPyWpdhFdJyxciw1p6ykpgtpjanoVGr19QKaEKEjeElUWN8//HF+HQ8oIHCE9Xo+Hufx6Ky99t6f7Tp63q19OQ5jjBEAAICFvDxdAAAAgKcQhAAAgLUIQgAAwFoEIQAAYC2CEAAAsBZBCAAAWIsgBAAArEUQAgAA1iIIAQAAaxGEYB2Hw1Gp18cff3xB+5kxY4YcDkfNFP3/OBwO3X///TW6zfrurbfeksPh0MKFCyvsk5GRIYfDoblz51Z6u2PGjFG7du1c2tq1a6cxY8acd92PP/642p+hDRs2aMaMGTp8+LDbsmuvvVbXXnttlbd5ofbu3SuHw6Gnn366zvcN1DYfTxcA1LXPP//c5f0TTzyhjz76SP/85z9d2rt27XpB+xk/frxuvPHGC9oGzm/w4MFq2bKl0tPTlZycXG6fxYsXy9fXV6NGjbqgfa1atUohISEXtI3z2bBhg2bOnKkxY8aocePGLssWLFhQq/sGbEQQgnWuuuoql/fNmzeXl5eXW/vZjh8/rqCgoErvp02bNmrTpk21akTl+fj4KCkpSXPmzNHXX3+t6Ohol+WHDx/WqlWrdPPNN6t58+YXtK8rrrjigta/UBcazgG449QYUI5rr71W0dHR+vTTT5WQkKCgoCDdfffdkqTly5crMTFRERERCgwMVJcuXTRlyhQdO3bMZRvlnRpr166dbrrpJr333nvq2bOnAgMD1blzZ6Wnp9dY7YcOHdKECRPUunVr+fn5qX379po2bZqKi4td+q1YsUJxcXEKDQ1VUFCQ2rdv7zxGSSotLdXs2bPVqVMnBQYGqnHjxurevbuee+65Cvf9008/yc/PT3/4wx/clu3cuVMOh0Pz5s2TdCZYTp48WVFRUQoICFDTpk0VGxur1157rcrHPG7cOElnZn7O9tprr+nEiRPOY5s/f76uueYatWjRQsHBwfr1r3+tOXPm6NSpU+fdT3mnxnbu3Kkbb7xRQUFBatasmZKTk3XkyBG3dTMyMnTLLbeoTZs2CggI0GWXXaZ77rlHBQUFzj4zZszQww8/LEmKiopyO01b3qmxyo532WnVv/71r+rSpYuCgoLUo0cPvf322+c97srKycnRXXfdpRYtWsjf319dunTRM888o9LSUpd+aWlp6tGjh371q1+pUaNG6ty5sx577DHn8pr8bADnw4wQUIHc3FzdddddeuSRR/SnP/1JXl5n/r/h+++/16BBgzRx4kQFBwdr586devLJJ7Vx40a302vl2bZtm373u99pypQpCg8P1yuvvKJx48bpsssu0zXXXHNBNZ84cUL9+vXTrl27NHPmTHXv3l2ZmZlKTU3V1q1b9c4770g6c3pwxIgRGjFihGbMmKGAgABlZ2e71D9nzhzNmDFDv//973XNNdfo1KlT2rlzZ7nXrpRp3ry5brrpJv3lL3/RzJkznX9m0pmQ4ufnpzvvvFOSlJKSor/+9a+aPXu2rrjiCh07dkxff/21Dh48WOXj7tixo66++mr97W9/05///Gf5+vq67Ld169a64YYbJEm7du3SyJEjFRUVJT8/P23btk1//OMftXPnzioH0h9//FF9+/aVr6+vFixYoPDwcC1btqzc67h27dql+Ph4jR8/XqGhodq7d6/mzp2rq6++Wl999ZV8fX01fvx4HTp0SM8//7xWrlypiIgISRXPBFV2vMu88847+vLLLzVr1iz96le/0pw5c3Trrbfq22+/Vfv27at07Gf76aeflJCQoJMnT+qJJ55Qu3bt9Pbbb2vy5MnatWuX87Te66+/rgkTJuiBBx7Q008/LS8vL/3v//6vtm/f7txWTX42gPMygOVGjx5tgoODXdr69u1rJJkPP/zwnOuWlpaaU6dOmU8++cRIMtu2bXMumz59ujn7r1hkZKQJCAgw2dnZzraff/7ZNG3a1Nxzzz3nrVWSue+++ypcvnDhQiPJ/P3vf3dpf/LJJ40ks27dOmOMMU8//bSRZA4fPlzhtm666SZz+eWXn7ems61Zs8ZlX8YYc/r0adOqVSszbNgwZ1t0dLQZOnRolbdfkcWLFxtJZuXKlc62r7/+2kgy06ZNK3edkpISc+rUKbN06VLj7e1tDh065Fw2evRoExkZ6dI/MjLSjB492vn+0UcfNQ6Hw2zdutWl34ABA4wk89FHH5W737LPTXZ2tpFk3nrrLeeyp556ykgye/bscVuvb9++pm/fvs73lR1vY858dsLDw01RUZGzLS8vz3h5eZnU1NRy6yyzZ88eI8k89dRTFfaZMmWKkWT+9a9/ubTfe++9xuFwmG+//dYYY8z9999vGjdufM791fRnAzgXTo0BFWjSpImuu+46t/bdu3dr5MiRatmypby9veXr66u+fftKknbs2HHe7V5++eW65JJLnO8DAgLUsWNHZWdnX3DN//znPxUcHKzbbrvNpb3sdM6HH34oSbryyislScOHD9ff//53/fDDD27b6tWrl7Zt26YJEybo/fffV1FRUaVqGDhwoFq2bOlymur999/XgQMHXE699erVS++++66mTJmijz/+WD///HOVjvVsw4cPV6NGjVxmddLT0+VwODR27FhnW1ZWlm6++WaFhYU5xy8pKUklJSX67rvvqrTPjz76SN26dVOPHj1c2keOHOnWNz8/X8nJyWrbtq18fHzk6+uryMhISZX73JSnsuNdpl+/fmrUqJHzfXh4uFq0aFFjn72uXbuqV69ebrUYY5yzjb169dLhw4d1xx136K233nI5NVimpj8bwLkQhIAKlJ2W+KWjR4+qT58++te//qXZs2fr448/1pdffqmVK1dKUqX+wQ4LC3Nr8/f3r5F/7A8ePKiWLVu6XZvUokUL+fj4OE8tXHPNNVq9erVOnz6tpKQktWnTRtHR0S7XYEydOlVPP/20vvjiCw0cOFBhYWHq37+/Nm3adM4afHx8NGrUKK1atcp5Gm3JkiWKiIhwnp6SpHnz5unRRx/V6tWr1a9fPzVt2lRDhw7V999/X61jDwoK0u2336733ntPeXl5On36tP72t7+pb9++uvTSSyWduYalT58++uGHH/Tcc88pMzNTX375pebPny+pcuP3S2V/3mc7u620tFSJiYlauXKlHnnkEX344YfauHGjvvjii2rt9+z9n2+8y9T2Z6+8vzOtWrVyLpekUaNGKT09XdnZ2Ro2bJhatGihuLg4ZWRkONep6c8GcC4EIaAC5T0D6J///KcOHDig9PR0jR8/Xtdcc41iY2Nd/i/bk8LCwvTjjz/KGOPSnp+fr9OnT6tZs2bOtltuuUUffvihCgsL9fHHH6tNmzYaOXKk8/ECPj4+SklJ0ZYtW3To0CG99tpr2rdvn2644QYdP378nHWMHTtWJ06c0Ouvv67//Oc/WrNmjZKSkuTt7e3sExwcrJkzZ2rnzp3Ky8tTWlqavvjiCw0ZMqTaxz9u3DidPn1aS5cu1dtvv638/HznhdSStHr1ah07dkwrV67UXXfdpauvvlqxsbHy8/Or1v7CwsKUl5fn1n5229dff61t27bpqaee0gMPPKBrr71WV155ZbnBpKr7r+x417awsDDl5ua6tR84cECSXGoZO3asNmzYoMLCQr3zzjsyxuimm25yzkzVxmcDqAhBCKiCsnDk7+/v0v7iiy96ohw3/fv319GjR7V69WqX9qVLlzqXn83f3199+/bVk08+KenMqaOzNW7cWLfddpvuu+8+HTp0SHv37j1nHV26dFFcXJwWL16sV199VcXFxS6np84WHh6uMWPG6I477tC333573qBVkbi4OEVHR2vx4sVavHixQkNDNWzYMOfy8sbPGKOXX365Wvvr16+fvvnmG23bts2l/dVXX3V5X5XPTVmfyszSVGe8a0v//v21fft2bdmyxa0Wh8Ohfv36ua0THBysgQMHatq0aTp58qS++eYbtz419dkAKsJdY0AVJCQkqEmTJkpOTtb06dPl6+urZcuWuX0R1qZdu3bpjTfecGvv2rWrkpKSNH/+fI0ePVp79+7Vr3/9a3322Wf605/+pEGDBun666+XJD3++OPav3+/+vfvrzZt2ujw4cN67rnnXK53GjJkiKKjoxUbG6vmzZsrOztbzz77rCIjI9WhQ4fz1nn33Xfrnnvu0YEDB5SQkKBOnTq5LI+Li9NNN92k7t27q0mTJtqxY4f++te/Kj4+3vm8pqVLl+ruu+9Wenq6kpKSKvXnc/fddyslJUXffvut7rnnHgUGBjqXDRgwQH5+frrjjjv0yCOP6MSJE0pLS9N//vOfSm37bBMnTlR6eroGDx6s2bNnO+8a27lzp0u/zp0769JLL9WUKVNkjFHTpk31j3/8w+V0UJlf//rXkqTnnntOo0ePlq+vrzp16lTurGNlx7umfPXVV+V+9q688kpNmjRJS5cu1eDBgzVr1ixFRkbqnXfe0YIFC3TvvfeqY8eOkqT//u//VmBgoHr37q2IiAjl5eUpNTVVoaGhzmvXKvPZAGqMJ6/UBi4GFd011q1bt3L7b9iwwcTHx5ugoCDTvHlzM378eLNlyxYjySxevNjZr6K7xgYPHuy2zbPvBqqIpApf06dPN8YYc/DgQZOcnGwiIiKMj4+PiYyMNFOnTjUnTpxwbuftt982AwcONK1btzZ+fn6mRYsWZtCgQSYzM9PZ55lnnjEJCQmmWbNmxs/Pz1xyySVm3LhxZu/eveet0xhjCgsLTWBgoJFkXn75ZbflU6ZMMbGxsaZJkybG39/ftG/f3kyaNMkUFBQ4+5TdCfbLP9fz+emnn4yfn5+RZDZu3Oi2/B//+Ifp0aOHCQgIMK1btzYPP/yweffdd93u8qrMXWPGGLN9+3YzYMAAExAQYJo2bWrGjRtn3nrrLbftlfVr1KiRadKkifmv//ovk5OT4zJ2ZaZOnWpatWplvLy8XLZT3uekMuNtTMV3HJZ3TGcru2usolfZ+GRnZ5uRI0easLAw4+vrazp16mSeeuopU1JS4tzWX/7yF9OvXz8THh5u/Pz8TKtWrczw4cPNv//9b2efynw2gJriMOask8sAAACW4BohAABgLYIQAACwFkEIAABYy6NB6NNPP9WQIUPUqlUrORwOt1tAy/PJJ58oJiZGAQEBat++vRYuXFj7hQIAgAbJo0Ho2LFj6tGjh1544YVK9d+zZ48GDRqkPn36KCsrS4899pgefPBBvfnmm7VcKQAAaIgumrvGHA6HVq1apaFDh1bY59FHH9WaNWtcfpcnOTlZ27Ztcz4NFwAAoLLq1QMVP//8cyUmJrq03XDDDVq0aJFOnTolX19ft3WKi4tVXFzsfF9aWqpDhw4pLCys3J9QAAAAFx9jjI4cOaJWrVrJy6vmTmjVqyCUl5en8PBwl7bw8HCdPn1aBQUF5f7gX2pqqmbOnFlXJQIAgFq0b98+tWnTpsa2V6+CkOT+Q5hlZ/Yqmt2ZOnWqUlJSnO8LCwt1ySWXaN++fQoJCam9QgEAQI0pKipS27Zta/xHrutVEGrZsqXbrzrn5+fLx8enwl9x9vf3d/uhQ0kKCQkhCAEAUM/U9GUt9eo5QvHx8W4/Urhu3TrFxsaWe30QAADAuXg0CB09elRbt27V1q1bJZ25PX7r1q3KycmRdOa01i9/cTo5OVnZ2dlKSUnRjh07lJ6erkWLFmny5MmeKB8AANRzHj01tmnTJvXr18/5vuxantGjR2vJkiXKzc11hiJJioqK0tq1azVp0iTNnz9frVq10rx58zRs2LA6rx0AANR/F81zhOpKUVGRQkNDVVhYyDVCAADUE7X1/V2vrhECAACoSQQhAABgLYIQAACwFkEIAABYiyAEAACsRRACAADWIggBAABrEYQAAIC1CEIAAMBaBCEAAGAtghAAALAWQQgAAFiLIAQAAKxFEAIAANYiCAEAAGsRhAAAgLUIQgAAwFoEIQAAYC2CEAAAsBZBCAAAWIsgBAAArEUQAgAA1iIIAQAAaxGEAACAtQhCAADAWgQhAABgLYIQAACwFkEIAABYiyAEAACsRRACAADWIggBAABrEYQAAIC1CEIAAMBaBCEAAGAtghAAALAWQQgAAFiLIAQAAKxFEAIAANYiCAEAAGsRhAAAgLUIQgAAwFoEIQAAYC2CEAAAsBZBCAAAWIsgBAAArEUQAgAA1iIIAQAAaxGEAACAtQhCAADAWgQhAABgLYIQAACwFkEIAABYiyAEAACsRRACAADWIggBAABrEYQAAIC1CEIAAMBaBCEAAGAtghAAALAWQQgAAFiLIAQAAKxFEAIAANYiCAEAAGt5PAgtWLBAUVFRCggIUExMjDIzM8/Zf9myZerRo4eCgoIUERGhsWPH6uDBg3VULQAAaEg8GoSWL1+uiRMnatq0acrKylKfPn00cOBA5eTklNv/s88+U1JSksaNG6dvvvlGK1as0Jdffqnx48fXceUAAKAh8GgQmjt3rsaNG6fx48erS5cuevbZZ9W2bVulpaWV2/+LL75Qu3bt9OCDDyoqKkpXX3217rnnHm3atKmOKwcAAA2Bx4LQyZMntXnzZiUmJrq0JyYmasOGDeWuk5CQoP3792vt2rUyxujHH3/UG2+8ocGDB1e4n+LiYhUVFbm8AAAAJA8GoYKCApWUlCg8PNylPTw8XHl5eeWuk5CQoGXLlmnEiBHy8/NTy5Yt1bhxYz3//PMV7ic1NVWhoaHOV9u2bWv0OAAAQP3l8YulHQ6Hy3tjjFtbme3bt+vBBx/U448/rs2bN+u9997Tnj17lJycXOH2p06dqsLCQudr3759NVo/AACov3w8teNmzZrJ29vbbfYnPz/fbZaoTGpqqnr37q2HH35YktS9e3cFBwerT58+mj17tiIiItzW8ff3l7+/f80fAAAAqPc8NiPk5+enmJgYZWRkuLRnZGQoISGh3HWOHz8uLy/Xkr29vSWdmUkCAACoCo+eGktJSdErr7yi9PR07dixQ5MmTVJOTo7zVNfUqVOVlJTk7D9kyBCtXLlSaWlp2r17t9avX68HH3xQvXr1UqtWrTx1GAAAoJ7y2KkxSRoxYoQOHjyoWbNmKTc3V9HR0Vq7dq0iIyMlSbm5uS7PFBozZoyOHDmiF154Qb/73e/UuHFjXXfddXryySc9dQgAAKAecxjLzikVFRUpNDRUhYWFCgkJ8XQ5AACgEmrr+9vjd40BAAB4CkEIAABYiyAEAACsRRACAADWIggBAABrEYQAAIC1CEIAAMBaBCEAAGAtghAAALAWQQgAAFiLIAQAAKxFEAIAANYiCAEAAGsRhAAAgLUIQgAAwFoEIQAAYC2CEAAAsBZBCAAAWIsgBAAArEUQAgAA1iIIAQAAaxGEAACAtQhCAADAWgQhAABgLYIQAACwFkEIAABYiyAEAACsRRACAADWIggBAABrEYQAAIC1CEIAAMBaBCEAAGAtghAAALAWQQgAAFiLIAQAAKxFEAIAANYiCAEAAGsRhAAAgLUIQgAAwFoEIQAAYC2CEAAAsBZBCAAAWIsgBAAArEUQAgAA1iIIAQAAaxGEAACAtQhCAADAWgQhAABgLYIQAACwFkEIAABYiyAEAACsRRACAADWIggBAABrEYQAAIC1CEIAAMBaBCEAAGAtghAAALAWQQgAAFiLIAQAAKxFEAIAANYiCAEAAGsRhAAAgLUIQgAAwFoeD0ILFixQVFSUAgICFBMTo8zMzHP2Ly4u1rRp0xQZGSl/f39deumlSk9Pr6NqAQBAQ+LjyZ0vX75cEydO1IIFC9S7d2+9+OKLGjhwoLZv365LLrmk3HWGDx+uH3/8UYsWLdJll12m/Px8nT59uo4rBwAADYHDGGM8tfO4uDj17NlTaWlpzrYuXbpo6NChSk1Ndev/3nvv6fbbb9fu3bvVtGnTau2zqKhIoaGhKiwsVEhISLVrBwAAdae2vr89dmrs5MmT2rx5sxITE13aExMTtWHDhnLXWbNmjWJjYzVnzhy1bt1aHTt21OTJk/Xzzz9XuJ/i4mIVFRW5vAAAACQPnhorKChQSUmJwsPDXdrDw8OVl5dX7jq7d+/WZ599poCAAK1atUoFBQWaMGGCDh06VOF1QqmpqZo5c2aN1w8AAOo/j18s7XA4XN4bY9zaypSWlsrhcGjZsmXq1auXBg0apLlz52rJkiUVzgpNnTpVhYWFzte+fftq/BgAAED95LEZoWbNmsnb29tt9ic/P99tlqhMRESEWrdurdDQUGdbly5dZIzR/v371aFDB7d1/P395e/vX7PFAwCABsFjM0J+fn6KiYlRRkaGS3tGRoYSEhLKXad37946cOCAjh496mz77rvv5OXlpTZt2tRqvQAAoOHx6KmxlJQUvfLKK0pPT9eOHTs0adIk5eTkKDk5WdKZ01pJSUnO/iNHjlRYWJjGjh2r7du369NPP9XDDz+su+++W4GBgZ46DAAAUE959DlCI0aM0MGDBzVr1izl5uYqOjpaa9euVWRkpCQpNzdXOTk5zv6/+tWvlJGRoQceeECxsbEKCwvT8OHDNXv2bE8dAgAAqMc8+hwhT+A5QgAA1D8N7jlCAAAAnkYQAgAA1iIIAQAAaxGEAACAtQhCAADAWgQhAABgLYIQAACwFkEIAABYiyAEAACsRRACAADWIggBAABrEYQAAIC1CEIAAMBaBCEAAGAtghAAALAWQQgAAFiLIAQAAKxFEAIAANaqVhDat2+f9u/f73y/ceNGTZw4US+99FKNFQYAAFDbqhWERo4cqY8++kiSlJeXpwEDBmjjxo167LHHNGvWrBotEAAAoLZUKwh9/fXX6tWrlyTp73//u6Kjo7Vhwwa9+uqrWrJkSU3WBwAAUGuqFYROnTolf39/SdIHH3ygm2++WZLUuXNn5ebm1lx1AAAAtahaQahbt25auHChMjMzlZGRoRtvvFGSdODAAYWFhdVogQAAALWlWkHoySef1Isvvqhrr71Wd9xxh3r06CFJWrNmjfOUGQAAwMXOYYwx1VmxpKRERUVFatKkibNt7969CgoKUosWLWqswJpWVFSk0NBQFRYWKiQkxNPlAACASqit7+9qzQj9/PPPKi4udoag7OxsPfvss/r2228v6hAEAADwS9UKQrfccouWLl0qSTp8+LDi4uL0zDPPaOjQoUpLS6vRAgEAAGpLtYLQli1b1KdPH0nSG2+8ofDwcGVnZ2vp0qWaN29ejRYIAABQW6oVhI4fP65GjRpJktatW6ff/OY38vLy0lVXXaXs7OwaLRAAAKC2VCsIXXbZZVq9erX27dun999/X4mJiZKk/Px8LkAGAAD1RrWC0OOPP67JkyerXbt26tWrl+Lj4yWdmR264oorarRAAACA2lLt2+fz8vKUm5urHj16yMvrTJ7auHGjQkJC1Llz5xotsiZx+zwAAPVPbX1/+1R3xZYtW6ply5bav3+/HA6HWrduzcMUAQBAvVKtU2OlpaWaNWuWQkNDFRkZqUsuuUSNGzfWE088odLS0pquEQAAoFZUa0Zo2rRpWrRokf785z+rd+/eMsZo/fr1mjFjhk6cOKE//vGPNV0nAABAjavWNUKtWrXSwoULnb86X+att97ShAkT9MMPP9RYgTWNa4QAAKh/Lqqf2Dh06FC5F0R37txZhw4duuCiAAAA6kK1glCPHj30wgsvuLW/8MIL6t69+wUXBQAAUBeqdY3QnDlzNHjwYH3wwQeKj4+Xw+HQhg0btG/fPq1du7amawQAAKgV1ZoR6tu3r7777jvdeuutOnz4sA4dOqTf/OY3+uabb7R48eKarhEAAKBWVPuBiuXZtm2bevbsqZKSkpraZI3jYmkAAOqfi+piaQAAgIaAIAQAAKxFEAIAANaq0l1jv/nNb865/PDhwxdSCwAAQJ2qUhAKDQ097/KkpKQLKggAAKCuVCkIcWs8AABoSLhGCAAAWIsgBAAArEUQAgAA1iIIAQAAaxGEAACAtQhCAADAWgQhAABgLYIQAACwFkEIAABYiyAEAACsRRACAADWIggBAABrEYQAAIC1CEIAAMBaBCEAAGAtghAAALCWx4PQggULFBUVpYCAAMXExCgzM7NS661fv14+Pj66/PLLa7dAAADQYHk0CC1fvlwTJ07UtGnTlJWVpT59+mjgwIHKyck553qFhYVKSkpS//7966hSAADQEDmMMcZTO4+Li1PPnj2VlpbmbOvSpYuGDh2q1NTUCte7/fbb1aFDB3l7e2v16tXaunVrpfdZVFSk0NBQFRYWKiQk5ELKBwAAdaS2vr89NiN08uRJbd68WYmJiS7tiYmJ2rBhQ4XrLV68WLt27dL06dMrtZ/i4mIVFRW5vAAAACQPBqGCggKVlJQoPDzcpT08PFx5eXnlrvP9999rypQpWrZsmXx8fCq1n9TUVIWGhjpfbdu2veDaAQBAw+Dxi6UdDofLe2OMW5sklZSUaOTIkZo5c6Y6duxY6e1PnTpVhYWFzte+ffsuuGYAANAwVG5apRY0a9ZM3t7ebrM/+fn5brNEknTkyBFt2rRJWVlZuv/++yVJpaWlMsbIx8dH69at03XXXee2nr+/v/z9/WvnIAAAQL3msRkhPz8/xcTEKCMjw6U9IyNDCQkJbv1DQkL01VdfaevWrc5XcnKyOnXqpK1btyouLq6uSgcAAA2Ex2aEJCklJUWjRo1SbGys4uPj9dJLLyknJ0fJycmSzpzW+uGHH7R06VJ5eXkpOjraZf0WLVooICDArR0AAKAyPBqERowYoYMHD2rWrFnKzc1VdHS01q5dq8jISElSbm7ueZ8pBAAAUF0efY6QJ/AcIQAA6p8G9xwhAAAATyMIAQAAaxGEAACAtQhCAADAWgQhAABgLYIQAACwFkEIAABYiyAEAACsRRACAADWIggBAABrEYQAAIC1CEIAAMBaBCEAAGAtghAAALAWQQgAAFiLIAQAAKxFEAIAANYiCAEAAGsRhAAAgLUIQgAAwFoEIQAAYC2CEAAAsBZBCAAAWIsgBAAArEUQAgAA1iIIAQAAaxGEAACAtQhCAADAWgQhAABgLYIQAACwFkEIAABYiyAEAACsRRACAADWIggBAABrEYQAAIC1CEIAAMBaBCEAAGAtghAAALAWQQgAAFiLIAQAAKxFEAIAANYiCAEAAGsRhAAAgLUIQgAAwFoEIQAAYC2CEAAAsBZBCAAAWIsgBAAArEUQAgAA1iIIAQAAaxGEAACAtQhCAADAWgQhAABgLYIQAACwFkEIAABYiyAEAACsRRACAADWIggBAABrEYQAAIC1CEIAAMBaBCEAAGAtghAAALAWQQgAAFjL40FowYIFioqKUkBAgGJiYpSZmVlh35UrV2rAgAFq3ry5QkJCFB8fr/fff78OqwUAAA2JR4PQ8uXLNXHiRE2bNk1ZWVnq06ePBg4cqJycnHL7f/rppxowYIDWrl2rzZs3q1+/fhoyZIiysrLquHIAANAQOIwxxlM7j4uLU8+ePZWWluZs69Kli4YOHarU1NRKbaNbt24aMWKEHn/88Ur1LyoqUmhoqAoLCxUSElKtugEAQN2qre9vj80InTx5Ups3b1ZiYqJLe2JiojZs2FCpbZSWlurIkSNq2rRphX2Ki4tVVFTk8gIAAJA8GIQKCgpUUlKi8PBwl/bw8HDl5eVVahvPPPOMjh07puHDh1fYJzU1VaGhoc5X27ZtL6huAADQcHj8YmmHw+Hy3hjj1lae1157TTNmzNDy5cvVokWLCvtNnTpVhYWFzte+ffsuuGYAANAw+Hhqx82aNZO3t7fb7E9+fr7bLNHZli9frnHjxmnFihW6/vrrz9nX399f/v7+F1wvAABoeDw2I+Tn56eYmBhlZGS4tGdkZCghIaHC9V577TWNGTNGr776qgYPHlzbZQIAgAbMYzNCkpSSkqJRo0YpNjZW8fHxeumll5STk6Pk5GRJZ05r/fDDD1q6dKmkMyEoKSlJzz33nK666irnbFJgYKBCQ0M9dhwAAKB+8mgQGjFihA4ePKhZs2YpNzdX0dHRWrt2rSIjIyVJubm5Ls8UevHFF3X69Gndd999uu+++5zto0eP1pIlS+q6fAAAUM959DlCnsBzhAAAqH8a3HOEAAAAPI0gBAAArEUQAgAA1iIIAQAAaxGEAACAtQhCAADAWgQhAABgLYIQAACwFkEIAABYiyAEAACsRRACAADWIggBAABrEYQAAIC1CEIAAMBaBCEAAGAtghAAALAWQQgAAFiLIAQAAKxFEAIAANYiCAEAAGsRhAAAgLUIQgAAwFoEIQAAYC2CEAAAsBZBCAAAWIsgBAAArEUQAgAA1iIIAQAAaxGEAACAtQhCAADAWgQhAABgLYIQAACwFkEIAABYiyAEAACsRRACAADWIggBAABrEYQAAIC1CEIAAMBaBCEAAGAtghAAALAWQQgAAFiLIAQAAKxFEAIAANYiCAEAAGsRhAAAgLUIQgAAwFoEIQAAYC2CEAAAsBZBCAAAWIsgBAAArEUQAgAA1iIIAQAAaxGEAACAtQhCAADAWgQhAABgLYIQAACwFkEIAABYiyAEAACsRRACAADWIggBAABrEYQAAIC1CEIAAMBaHg9CCxYsUFRUlAICAhQTE6PMzMxz9v/kk08UExOjgIAAtW/fXgsXLqyjSgEAQEPj0SC0fPlyTZw4UdOmTVNWVpb69OmjgQMHKicnp9z+e/bs0aBBg9SnTx9lZWXpscce04MPPqg333yzjisHAAANgcMYYzy187i4OPXs2VNpaWnOti5dumjo0KFKTU116//oo49qzZo12rFjh7MtOTlZ27Zt0+eff16pfRYVFSk0NFSFhYUKCQm58IMAAAC1rra+vz02I3Ty5Elt3rxZiYmJLu2JiYnasGFDuet8/vnnbv1vuOEGbdq0SadOnaq1WgEAQMPk46kdFxQUqKSkROHh4S7t4eHhysvLK3edvLy8cvufPn1aBQUFioiIcFunuLhYxcXFzveFhYWSziRLAABQP5R9b9f0iSyPBaEyDofD5b0xxq3tfP3Lay+TmpqqmTNnurW3bdu2qqUCAAAPO3jwoEJDQ2tsex4LQs2aNZO3t7fb7E9+fr7brE+Zli1bltvfx8dHYWFh5a4zdepUpaSkON8fPnxYkZGRysnJqdE/SFRPUVGR2rZtq3379nHNlocxFhcPxuLiwVhcPAoLC3XJJZeoadOmNbpdjwUhPz8/xcTEKCMjQ7feequzPSMjQ7fccku568THx+sf//iHS9u6desUGxsrX1/fctfx9/eXv7+/W3toaCgf6otISEgI43GRYCwuHozFxYOxuHh4edXs5c0evX0+JSVFr7zyitLT07Vjxw5NmjRJOTk5Sk5OlnRmNicpKcnZPzk5WdnZ2UpJSdGOHTuUnp6uRYsWafLkyZ46BAAAUI959BqhESNG6ODBg5o1a5Zyc3MVHR2ttWvXKjIyUpKUm5vr8kyhqKgorV27VpMmTdL8+fPVqlUrzZs3T8OGDfPUIQAAgHrM4xdLT5gwQRMmTCh32ZIlS9za+vbtqy1btlR7f/7+/po+fXq5p8tQ9xiPiwdjcfFgLC4ejMXFo7bGwqMPVAQAAPAkj//WGAAAgKcQhAAAgLUIQgAAwFoEIQAAYK0GGYQWLFigqKgoBQQEKCYmRpmZmefs/8knnygmJkYBAQFq3769Fi5cWEeVNnxVGYuVK1dqwIABat68uUJCQhQfH6/333+/Dqtt+Kr6d6PM+vXr5ePjo8svv7x2C7RIVceiuLhY06ZNU2RkpPz9/XXppZcqPT29jqpt2Ko6FsuWLVOPHj0UFBSkiIgIjR07VgcPHqyjahuuTz/9VEOGDFGrVq3kcDi0evXq865TI9/fpoF5/fXXja+vr3n55ZfN9u3bzUMPPWSCg4NNdnZ2uf13795tgoKCzEMPPWS2b99uXn75ZePr62veeOONOq684anqWDz00EPmySefNBs3bjTfffedmTp1qvH19TVbtmyp48obpqqOR5nDhw+b9u3bm8TERNOjR4+6KbaBq85Y3HzzzSYuLs5kZGSYPXv2mH/9619m/fr1dVh1w1TVscjMzDReXl7mueeeM7t37zaZmZmmW7duZujQoXVcecOzdu1aM23aNPPmm28aSWbVqlXn7F9T398NLgj16tXLJCcnu7R17tzZTJkypdz+jzzyiOncubNL2z333GOuuuqqWqvRFlUdi/J07drVzJw5s6ZLs1J1x2PEiBHm97//vZk+fTpBqIZUdSzeffddExoaag4ePFgX5VmlqmPx1FNPmfbt27u0zZs3z7Rp06bWarRRZYJQTX1/N6hTYydPntTmzZuVmJjo0p6YmKgNGzaUu87nn3/u1v+GG27Qpk2bdOrUqVqrtaGrzlicrbS0VEeOHKnxH9izUXXHY/Hixdq1a5emT59e2yVaozpjsWbNGsXGxmrOnDlq3bq1OnbsqMmTJ+vnn3+ui5IbrOqMRUJCgvbv36+1a9fKGKMff/xRb7zxhgYPHlwXJeMXaur72+NPlq5JBQUFKikpcfv1+vDwcLdfrS+Tl5dXbv/Tp0+roKBAERERtVZvQ1adsTjbM888o2PHjmn48OG1UaJVqjMe33//vaZMmaLMzEz5+DSofyo8qjpjsXv3bn322WcKCAjQqlWrVFBQoAkTJujQoUNcJ3QBqjMWCQkJWrZsmUaMGKETJ07o9OnTuvnmm/X888/XRcn4hZr6/m5QM0JlHA6Hy3tjjFvb+fqX146qq+pYlHnttdc0Y8YMLV++XC1atKit8qxT2fEoKSnRyJEjNXPmTHXs2LGuyrNKVf5ulJaWyuFwaNmyZerVq5cGDRqkuXPnasmSJcwK1YCqjMX27dv14IMP6vHHH9fmzZv13nvvac+ePc4fC0fdqonv7wb1v3nNmjWTt7e3W5LPz893S41lWrZsWW5/Hx8fhYWF1VqtDV11xqLM8uXLNW7cOK1YsULXX399bZZpjaqOx5EjR7Rp0yZlZWXp/vvvl3Tmy9gYIx8fH61bt07XXXddndTe0FTn70ZERIRat26t0NBQZ1uXLl1kjNH+/fvVoUOHWq25oarOWKSmpqp37956+OGHJUndu3dXcHCw+vTpo9mzZ3MWoQ7V1Pd3g5oR8vPzU0xMjDIyMlzaMzIylJCQUO468fHxbv3XrVun2NhY+fr61lqtDV11xkI6MxM0ZswYvfrqq5xzr0FVHY+QkBB99dVX2rp1q/OVnJysTp06aevWrYqLi6ur0huc6vzd6N27tw4cOKCjR48627777jt5eXmpTZs2tVpvQ1adsTh+/Li8vFy/Or29vSX9/9kI1I0a+/6u0qXV9UDZrZCLFi0y27dvNxMnTjTBwcFm7969xhhjpkyZYkaNGuXsX3b73aRJk8z27dvNokWLuH2+hlR1LF599VXj4+Nj5s+fb3Jzc52vw4cPe+oQGpSqjsfZuGus5lR1LI4cOWLatGljbrvtNvPNN9+YTz75xHTo0MGMHz/eU4fQYFR1LBYvXmx8fHzMggULzK5du8xnn31mYmNjTa9evTx1CA3GkSNHTFZWlsnKyjKSzNy5c01WVpbzUQa19f3d4IKQMcbMnz/fREZGGj8/P9OzZ0/zySefOJeNHj3a9O3b16X/xx9/bK644grj5+dn2rVrZ9LS0uq44oarKmPRt29fI8ntNXr06LovvIGq6t+NXyII1ayqjsWOHTvM9ddfbwIDA02bNm1MSkqKOX78eB1X3TBVdSzmzZtnunbtagIDA01ERIS58847zf79++u46obno48+Oud3QG19fzuMYS4PAADYqUFdIwQAAFAVBCEAAGAtghAAALAWQQgAAFiLIAQAAKxFEAIAANYiCAEAAGsRhABYyeFwaPXq1Z4uA4CHEYQA1LkxY8bI4XC4vW688UZPlwbAMg3q1+cB1B833nijFi9e7NLm7+/voWoA2IoZIQAe4e/vr5YtW7q8mjRpIunMaau0tDQNHDhQgYGBioqK0ooVK1zW/+qrr3TdddcpMDBQYWFh+u1vf+vy6+ySlJ6erm7dusnf318RERG6//77XZYXFBTo1ltvVVBQkDp06KA1a9Y4l/3nP//RnXfeqebNmyswMFAdOnRwC24A6j+CEICL0h/+8AcNGzZM27Zt01133aU77rhDO3bskCQdP35cN954o5o0aaIvv/xSK1as0AcffOASdNLS0nTffffpt7/9rb766iutWbNGl112mcs+Zs6cqeHDh+vf//63Bg0apDvvvFOHDh1y7n/79u169913tWPHDqWlpalZs2Z19wcAoG5c8M/FAkAVjR492nh7e5vg4GCX16xZs4wxxkgyycnJLuvExcWZe++91xhjzEsvvWSaNGlijh496lz+zjvvGC8vL5OXl2eMMaZVq1Zm2rRpFdYgyfz+9793vj969KhxOBzm3XffNcYYM2TIEDN27NiaOWAAFy2uEQLgEf369VNaWppLW9OmTZ3/HR8f77IsPj5eW7dulSTt2LFDPXr0UHBwsHN57969VVpaqm+//VYOh0MHDhxQ//79z1lD9+7dnf8dHBysRo0aKT8/X5J07733atiwYdqyZYsSExM1dOhQJSQkVOtYAVy8CEIAPCI4ONjtVNX5OBwOSZIxxvnf5fUJDAys1PZ8fX3d1i0tLZUkDRw4UNnZ2XrnnXf0wQcfqH///rrvvvv09NNPV6lmABc3rhECcFH64osv3N537txZktS1a1dt3bpVx44dcy5fv369vLy81LFjRzVq1Ejt2rXThx9+eEE1NG/eXGPGjNHf/vY3Pfvss3rppZcuaHsALj7MCAHwiOLiYuXl5bm0+fj4OC9IXrFihWJjY3X11Vdr2bJl2rhxoxYtWiRJuvPOOzV9+nSNHj1aM2bM0E8//aQHHnhAo0aNUnh4uCRpxowZSk5OVosWLTRw4EAdOXJE69ev1wMPPFCp+h5//HHFxMSoW7duKi4u1ttvv60uXbrU4J8AgIsBQQiAR7z33nuKiIhwaevUqZN27twp6cwdXa+//romTJigli1batmyZerataskKSgoSO+//74eeughXXnllQoKCtKwYcM0d+5c57ZGjx6tEydO6H/+5380efJkNWvWTLfddlul6/Pz89PUqVO1d+9eBQYGqk+fPnr99ddr4MgBXEwcxhjj6SIA4JccDodWrVqloUOHeroUAA0c1wgBAABrEYQAAIC1uEYIwEWHM/YA6gozQgAAwFoEIQAAYC2CEAAAsBZBCAAAWIsgBAAArEUQAgAA1iIIAQAAaxGEAACAtQhCAADAWv8H/ybPZSF9fRYAAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "plt.title(\"Train Loss vs. Validation Loss\")\n",
        "plt.xlabel(\"Epochs\")\n",
        "plt.ylabel(\"Loss\")\n",
        "\n",
        "plt.plot(range(EPOCHS), train_losses, label=\"Train Loss\")\n",
        "plt.plot(range(EPOCHS), val_losses, label=\"Val Loss\")\n",
        "\n",
        "plt.legend()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e6f47116-c4d1-4dfb-a13c-e9fe2bf9f52d",
      "metadata": {},
      "outputs": [],
      "source": [
        "_l"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "base",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.11"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}
